YellowFin and the Art of Momentum Tuning

Adaptive Optimization Hyperparameter tuning is a big cost of deep learning. Momentum: a key hyperparameter to SGD and variants. Adaptive methods, e.g. Adam1, don’t tune momentum. YellowFin optimizer • Based on the robustness properties of momentum. • Auto-tuning of momentum and learning rate in SGD. • Closed-loop momentum control for async. training. Experiments ResNet and LSTM YellowFin runs with no tuning. Adam, mom. SGD etc. are tuned on learning rate grids. YellowFin can outperform tuned SoA on train/val. metrics.